The ZPAQ Compression Algorithm

ZPAQ is a tool for creating compressed archives and encrypted user-level incremental backups with rollback capability. It deduplicates any new or modified files by splitting them into fragments along content-dependent boundaries and comparing their cryptographic hashes to previously stored fragments. Unmatched fragments are grouped by file type and packed into blocks and either stored or compressed independently in multiple threads using LZ77, BWT, or context mixing in a self-describing format depending on the user selected compression level and an analysis of the input data. Speed and compression ratio compare favorably with other archivers. Introduction ZPAQ [1] is a tool for producing compressed archives and user-level incremental backups. Figure 1 compares ZPAQ with some popular archivers and backup utilities at default and maximum compression settings on on the 10GB corpus [2], a set of 79K files in 4K directories. Figure 1. 10 GB compression speed and size at default and maximum settings. Compression plus decompression times are in real seconds for 10 GB on a 2.67 GHz Core i7 M620, 2+2 hyperthreads, 4 GB memory under Ubuntu Linux with the archive on an external USB drive. The 10 GB corpus consists of 79K files in 4K directories with a wide range of file types. It includes about 2 GB of already compressed files (ZIP, JPEG, etc), and 1.5 GB of duplicate data to simulate realistic backup scenarios. ZIP [3], 7ZIP [4], RAR [5], and FREEARC [6] are primarily archivers. The compression algorithms are optimized for fast decompression, typically some variant of LZ77 [7]. They do not support deduplication or rollback. ZIP compresses each file separately. The others have a “solid” mode which compresses better but requires decompressing all files in order to just extract one or to update. ZIP and RAR support incremental updates based on changes to file last-modified dates. ZPAQ, PCOMPRESS [8], EXDUPE [9], and OBNAM [10] are primarily backup utilities. The compression algorithms are optimized for fast compression rather than decompression. All of them support deduplication by storing cryptographic hashes of file fragments split along content dependent boundaries. If two fragments have the same hash then they are assumed to be identical and stored only once. All but PCOMPRESS support incremental updates. PCOMPRESS archives cannot be updated at all. PCOMPRESS and OBNAM run only in Unix or Linux. OBNAM produces a large directory tree rather than a single archive file. ZPAQ and OBNAM can be reverted to an earlier state to extract old versions of files that have since been modified. ZPAQ has one of its 5 compression levels (-method 2) with fast decompression to support its use as an archiver. PCOMPRESS has the best overall compression performance in the middle range, and ZPAQ in the low and high range. Table 1 shows the data from Figure 1 with separate compression and decompression times. Times with a * are on the Pareto frontier (no program is faster and compresses better). Table 1. 10GB corpus compression and decompression times in real seconds. Size Compress Extract Program Options ----------------------------------------------2788126729 20291* 19898* zpaq 6.51 -m 5 2893742274 1756* 838* pcompress 3.1 -l14 -s6